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ABSTRACT 



The goal of the evaluation of the Western Region 
AMIDS was to probe deeper into the content, organization, management 
and methodology of the program to determine its impact on the adult, 
vocational, technical, and manpower programs that are served by it. 
The theories and methods developed by Edward Suchman and Robert F. 
Stake were selected as being most applicable. Identification of 
specific aspects of Western AMIDS to be evaluated was facilitated by 
the use of Suchman' s five categories of criteria for program 
evaluation--effort, performance, adequacy of performance, efficiency, 
and process. The functional flow of work was divided into three 
stages; Stage one-- identification of goals, intents and assumptions 
by reviewing literature and legislation, and interviewing program 
staff and the program administrators; State two--relating the results 
of the first stage to Suchman's five evaluative criteria; Stage 
three--comparing observed inputs, transactions, and outputs for each 
criteria with two outside standards. Two major types of reporting, 
formal reports and informal feedback, were planned. An illustrative 
example of the evaluative process, and a schedule of time and 
activities are provided. (PT) 
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FOREWORD 



Evaluation has become an integral part of vocational education and manpower 
programs. The evaluation plan for Western Region AMIDS departs considerably 
from traditional evaluative processes and utilizes a modern, and tested, 
theoretical structure and methodology. 

The five categories of criteria for evaluation of the AMIDS program are: 

Effort, Performance, Adequacy of Performance, Efficiency, and Process. 

Data related to these criteria will be obtained "on-site" from a realistic 
cross-section of the work performed by Western Region AMIDS during the year 
1970-71. 

The Evaluation Team is under the direction of Dr. James A. Farmer ,'Jr., 
Assistant Professor, Graduate School of Education, University of California, 

Los Angeles. Dr. Farmer is assisted in the evaluation program by two 
Research Specialists of the Division of Vocational Education, Ralph k. 

Sylvester and Patrick J. Weagraff, who are candidates for the degree 
Doctor of Education, UCLA. Mr. Manuel Jimenes, Director of Western Region 
AMIDS and Mr. Lee W. Ralston, Chief Consultant, provide program information 
and guidance for the Evaluation Team. 

Melvin L. Barlow 

Director, Division of Vocational Education 
University of California 

Professor of Education 
Graduate School of Education 
University of California, Los Angeles 
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INTRODUCTION 



The process of evaluating the Western Region AMIDS Project is not 
a simple one because it deals with the measurement of human behavior as 
well as the effects of human behavior. Further difficulty arises from 
the on-going, ever-changing nature of AMIDS efforts. With these problems 
in mind, during August 1970 the Western AMIDS evaluation team gathered 
basic data concerning AMIDS Western program. Concurrently, a research 
design was developed and this process report was prepared. 

As its first step, the team endeavored to plan an overall 
evaluative research design. Although this report primarily presents 
the evaluation plan, it also describes the projected team activities. 

The Western AMIDS Evaluation Plan of July 22, 1970, states that 
the evaluation teami 
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",,,wlll probe deeper into the content, 
organization , management and methodology 
of the AHIDS program. This probe will 
determine the impact of the AMIDS program 
upon the adult, vocational , technical and 
manpower program which are served by AMIDS, 
Further, it will obtain evidence of behav- 
ioral change among the personnel who attei.d 
the AMIDS program. It is envisioned that 
the AMIDS Evaluation team will provide the 
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evidence which will support the general 
rationale that an AMIDS program is necessary 
and valuable and that it does in fact have 
a legitimate place in vocational education 
and manpower efforts of the nation . 

(.•.if this rationale is true , we should be 
able to prove it)." 



To carry out this responsibility effectively, the evaluation team 

examined a variety of approaches. After careful consideration, the 

theories and methods developed by two experts, Edward Suchman* and Robert 
2 

E. Stake* have been selected as being the most applicable. Their 
concepts of evaluative research are embodied in this plan. 

During the preliminary development of the project design, staff 
members of the Center for the Study of Esaluation at UCLA gave valuable 
assistance, and helpful suggestions were made by Mr. Lee Ralston, Chief 
Consultant to the Western Region AMIDS program. Mr. Ralston was instrumental 
in strengthening the effectiveness of the proposed approach. 
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PROPOSED RESEARCH DESIGN 

Continuing evaluation of in-service staff development too often 
consists merely of course examinations # the use of learner satisfaction 
forms and r. review of enrollment. By contrast# the approach of this 
evaluation is far more comprehensive in scope and content. It was designed 
to provide evaluative information about Western AMIDS which can be used 
for substantial program improvement. Further# such evaluation can be 
utilized to reduce duplication of effort and in the identification of 
emergent needs for Western .AMIDS . 

A. E VALUATION CATEGORIES 

AMIDS has numerous broad goals# one of which is the provision of 
long-term benefits to the MDT staff members who participate. For the 
purpose of evaluation# however# it is necessary to focus on achievements 
of specific program objectives which provide training for participants 
and. as a result# strengthen manpower programs in the region. Advantages 
which can accrue from the AMIDS program include more adequate institu- 
tional plant utilization# increased citizen support# and improved inter- 
institutional cooperation. An important part of the evaluation team's 
responsibility is the collecting and reporting of data concerning the 
extent to which the program benefits participants and their programs. 

It is admittedly impossible to collect data on every aim and out- 
come related to any functioning program# and for this reason evaluation 
efforts must be selective. Identification of specific aspects of Western 
AMIDS to bo evaluated was facilitated by the use of Suchman's five 
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categories of criteria for program evaluation. These categories can be 
summarized as follows: 

1. EFFORT . The criterion for success in this category is the 

quantity and quality of activity that ocvursj this 
represents an assessment of input or energy regard- 
less of output. It is intended to answer the questions 
"What did you do?" and "How vail did you do it?" 

2. PERFORMANCE . Performance or effect criteria measure the results 

of effort rather than the effort itself. Although this 
procedure requires a clear goal, it measures only how 
much is acconplished in relation to the goal. It is 
a response to the questions "Did any change occur?" 
and "Was the change the intended one?" 

3. ADEQUACY OF PERFORMANCE . Here the criterion of success is the 

degree to which effective performance is adequate to 
the total need. The evaluation team recognizes that 
this must be tempered by an awareness of realistic 
possibilities and available resources. 

4. EFFICIENCY . A positive reply to the question "Does it work?" 

often gives rise to another query, "Is there any 
better way to attain the same results?" The concept 
of efficiency involves the evaluation of alternative 
paths or methods in terms of costs - in money, time, 
personnel, etc. In a sense, it represents a ratio 
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between effort and performance - output divided by 
input . 

5. PROCESS . In the course of evaluating a program, it is expected 
that much can be learned about how and why a program 
works or fails to work. An analysis of process can 
provide administrative feedback as well as evidence 
for making both specific and general judgments about 
AMIDS. 

The evaluation team plans to deal with the process 
by analyzing: 

a) the attributes of the program's participants 

b) the attributes exposed to the program 

c) the situational context within which the 
program takes place 

d) the different kinds of effects produced 
hy the program. 

B. FUNCTIONAL FLOW OP WORK 

As demonstrated in Figure 1, this project consists of the following 

stages: 

Stage I 

Goals, intents, and underlying assumptions concerning Western AMIDS 
will be identified by: a) reviewing relevant literature and legislation: 

b) interviewing program staff: and e) interviewing administrators cf the 
program. 
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Preliminary Identification of Key 
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Stage II 

Results of the first stage will be related to Suchman's five 
evaluative criteria to generate key indicators for specific operational- 
ized objectives. 

Stage III 

In keeping with a process developed by Stake/ intended versus 
observed inputs/ transactions/ and outputs for each criterion will be 
compared in the light of two outside standards in the process of making 
specific judgments about Western AMDS. One of these outside standards 
is descriptive data from othar AMDS projects; the other consists of 
general standards of excellence. 

C. AN ILLUSTRATION OF THE EVALUATIVE PROCESS 

To illustrate the nature of this evaluative process/ the evalua- 
tion team identified one of Western AMDS’ central objectives/ namely/ 
to provide workshops for Manpower personnel. The way in which such 
workshops can be evaluated in terms of the above stages is described 
below. 

Stage I: Identifying Goals/ Intents and Assumptions 

The nature of the goals/ intents and assumptions underlying the 
AMDS program was sunmarized in May/ 1968 by Howard A. Matthews/ Director, 
Division of Manpower Development and Training/ Department of Health/ 
Education, and Welfare, who gave testimony before the House of Represent- 
ative's Committee on Education and Labor concerning the goals of the 
AMIDS program. He stated that AMDS has been established : 
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" ...for the express purpose of helping MDTA 
instructors and supervisors become more sen- 
sitive to the characteristics , problems and 
needs of disadvantaged trainees and develop 
instructional techniques and materials which 
will permit trainees to experience success . 
*AMIDS 9 will be extremely useful in orienting 
instructors who are occupationally competent, 
but who have never taught in a classroom, and 
in reorienting experienced vocational instruc- 
tors to MDTA trainees . Traditional teacher 
training institutions have not directed their 
training programs broadly enough to include 
the teacher of other than the middle-class 
student • The * AM1DS 9 are not only designed 
to meet an MDTA instructor training need, but 
will represent a modest pioneer effort in 
developing instructional techniques useful in 
teaching trainees with very diverse cultural 
backgrounds . Plans for 'AMIDS' assume a 
constant feed-back of information to the public 
school system . 

The 9 AMIDS 9 are hut one illustration of 
innovative possibilities of MDTA . We feel that 
these sites can do the most adequate job of 
preparing instructors for teaching the disad- 
vantaged out-of -school and out-of-work persons 
for the inner cities . They are also the ideal 
place to bring counselors for orientation from 
the public employment services and from other 
manpower programs, who must have a working know- 
ledge of the problems of the instructional 
process where the victimized, disadvantaged 
persons is concerned . " 
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^Current Objective; Providing Appropriate Workshops 

As part of its efforts to achieve such goals and on the 
basis of the type of program intents described in Dr. Matthews' 
speech, Western AMIDS has identified as one of its current objectives 
the provision of appropriate workshops for staff members of 
manpower programs in the region. 
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2. Underlying Assumptions 



It is assumed in offering these workshops that manpower 
staff members coming to AMIDS workshops constitute a group which 
can benefit from special attention. Also, it is assumed that 
they can more easily reach the target population for manpower 
programs than those who work in rented agencies, industry, 
business, as well as other potential participants who work with 
disadvantaged less frequently* This assumption requires frequent 
reexamination in view of the changing picture of manpower needs . 



Stage lit Evaluative Criteria Used to Generate Key Indicators 
In this stage, examples of what will be evaluated in the workshops 
are provided in relation to each of Suchman's five categories of criteria 
by which Western AMIDS 1 strengths and weaknesses will be evaluated. 

1. Effort (What did AMIDS do?) 

Number of workshops given, number of manpower programs 
involved, number of manpower staff who attended and 
completed workshop, proportion of attendees followed 
up with additional help, number of hours of technical 
assistance given in proportion to hours spent in 
program development. 

Sources of the Measures of Effort 
Review of records, interviews, etc. 

2. Performance (Did any change occur? If so, how much? 

That intended?) 

Degree of change in participants, yield of new applications 
for attendance at scheduled workshop, yield of new 
requests for technical assistance, yield for each 
category in the target population. 

Sources of the Measures of Performanc e 
Pre-test, post-test of participants on cognitive and/or 
attitudinal scales. Review of records to determine 
changes in request patterns, satisfaction questionnaires? 
interviewing participants . 
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3. 



Adequacy of Performance (Was the performance successful 

in terms of total need?) 

Total manpower programs that have been involved in 
workshops consistent with the known total need in the 
region* total past involvement consistent with the 
‘total potential need of industry, business, and other 
sources* total of new participants found in terms of 
the total estimated potential. 

Sources of the Measures of Adequacy of Performance 
Interviews and/or questionnaires to key administrators, 
AM1DS participants or other appropriate personnel. 

4. Efficiency (is there a better way to obtain the same 

results? Ratio of effort to performance*) 
Are the right clients being reached? Are there 
alternative methods in terms of costs? What is the 
learning efficiency? Could the applications for 
attendance be handled more efficiently? 

Sources of the Measures of Efficiency 
Records, interviews of administrators, observation 
and follow-up, comparison of different types of 
participants in terms of gains in performance. 

5. Process (Does the Program work?) 

Under what conditions and in what situations does the 
workshop proceed most effectively? Why? Among which 
groups? Are there any unintended negative consequences? 
Do the workshops or their evaluation reveal any other 
conditions? 



Sources of the Measures of Process 
Quasi-experiment or experiment (i.e., with control 
and experimental groups) . 



Stage Ills Comparison of Intentions and Results for Purpose of 
Making Judgments 

Finally, the key indicators of what is to be evaluated in the 
workshops will be compared in terms of intended versus observed inputs, 
transactions and outcomes. This process will permit specific judgments 
to be made about the workshops. 
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SCHEDULE OF TIME AND ACTIVITIES 



The evaluation of AMIDS Western Region program will require 
approximately ten months. As indicated in Figure 2, this ten-month 
period consists of four phases. Euch phase is three months in length, 
with a two-week overlap on the succeeding phase. 

During Phase 1, the project will be designed, work planned, and 
basic data gathered. The evaluation team feels that careful thought and 
pre-planning during this phase will greatly enhance the efficiency and 
effectiveness of all subsequent efforts. 

The primary concerns of Phases 2 and 3 are the analysis and syn- 
thesis of the collected information. However, since AMIDS is an ongoing 
program, some basic data will continue to be acquired. Throughout these 
phases attention will be devoted to developing a body of knowledge which 
is appropriate to ongoing and emergent decision- and conclusion-oriented 
demands. 

The fourth phase of the Western AMIDS evaluation calls for establish- 
ing and reporting on the causal relationship between AMIDS' intents and 
achievements. At the conclusion of Phase 4, generalizations and appro- 
priate recommendations will be disseminated. 
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Figure 2. AMIDS Evaluation Time Line 



REPORTS AND FEEDBACK 



Two major types of reporting are planned. They are* 

FORMA L REPORTS . As Figure 2 shows , two formal reports are 
envisaged. The first, an interim report, will be prepared 
for distribution by January 1, 1971. A second and final 
report will be sent out by June 30, 1971. Other formal 
reports may be developed as needed. 

2. INFORMAL FEEDBACK , a major objective of evaluation is 

program improvement, and provisions are therefore included 
for internal feedback to assure progress. Although this 
feedback will be given informally, a log and memorandum of 
conversation will also be kept. The evaluation staff 
assumes full responsibility for disseminating appropriate 
feedback data to the program staff. 

Both the formal reports and informal feedback will be 
handled in a differentiated manner, in other words, 
information most appropriate to specific persons and 
organizations will be directed to them to help strengthen 
the ongoing AMIDS efforts. 
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PROPOSED ACTIVITIES 
Phases 1 and 2 



( 

) 



The evaluation team proposes to engage in the evaluation of the following 
activities which represent a realistic cross-section of the work performed 
by Western Region AMIDS. 

On-Site Evaluation 



Workshop Date 

September 

21 - 25, 1970 



October 

5-9, 1970 



October 

1, 8, 22, 29, 1970 



November 
2-6, 1970 



December 

7 - 11, 1970 



Location 

Portland, Oregon 

Los Angeles, 

California 

Pasadena, 

California 

Tucson, Arizona 

San Francisco, 

California 



Team Member *nd 
Dates Attending 

Farmer, 21-22nd 
Sylvester, 23-24th 
Weagraff, 24-25th 



Fanner, 5,9th 
Sylvester, 6,8,9th 
Weagraff, 5,7,9th 



Weagraff, 8,22nd 



Sylvester, 4, 5, 6th 
Weagraff, 2,3,4th 



Fanner, 7,8th 



Additional data will be sought from the other AMIDS sites and programs 
providing similar training to permit relative comparisons. 




14 



\ 



REFEREMCES 



1. Suchman, Edward Evaluative Research , Russell Sage Foundation, 

New York, 1967. 

2. Stake, Robert E., "The Countenance of Educational Evaluation," 

Columbia University Teachers College Record 68« 

523-540, April 1967. 



ERIC Clearinghouse 

NOV 13 19/ u 

on Adult Education 



Seba Kolb, Editor 
9/28/70 



O 

ERIC 



15 



